“Gold Standard CX Metric”
As expressed by Qualtrics about NPS®, “NPS®
is often held up as the gold standard customer experience
metric.”
For the banking and financial services in particular, Cvetilena
Gocheva writes on the CustomerGauge website that this CX metric “has
never been more important”.
This project encompasses interactive NPS visualization and analysis.
I thank Kaggle and the contributor for the “dataset from
2021, generated using real distributions of NPS data from a retail
bank”. This Kaggle dataset is published under the Apache 2.0 open
source license.
My GitHub account contains
this and other projects as well as contact information. You are welcome
to contact me.
I am pleased to insert the attribution language requested by Bain
& Company, Inc.: “Net
Promoter®, NPS®, NPS Prism®, and the
NPS-related emoticons are registered trademarks of Bain & Company,
Inc., Satmetrix Systems, Inc., and Fred Reichheld. Net Promoter
Score℠ and Net Promoter System℠ are service marks
of Bain & Company, Inc., Satmetrix Systems, Inc., and Fred
Reichheld.”
TAGS: NPS interactive visualization, NPS analysis,
market segmentation, temporal analysis, cohort analysis, R, R Markdown,
HTML, PPTX
Data
Let us take a quick look at the input data in an exploratory data
analysis approach.
Dispersion
The heatmap chart below shows the NPS responses dispersion.
On the heatmap chart above, there is a strong response concentration
at the top of the chart – especially at level 10 and to a lesser extent
9 and 8 –, and there is some relatively dense concentration as well at
the zero level.
At any level, months are not populated evenly, but at levels 10, 9,
and 8 all months are populated in a relatively substantial measure. At
level 10, the responses number culminates at 154 in March and never
falls below 116 responses per month. At levels 9 and 8, the minimum per
month is 45.
By the way, at level zero, the minimum per month is still 35. Other
levels are less populated, and the responses number per month even falls
in one case at a figure as modest as 6.
Content
Except for scores, what do available input data provide as
information? We see it in the next interactive table.
In the table above, each NPS response is characterized among others
by
- the market where the customer banks,
- the response day,
- the customer’s name,
- the score.
From the Kaggle website, we learn that the question asked was “how likely
are you to recommend X to your friends and family?” and that “This a
dataset from 2021, generated using real distributions of NPS data from a
retail bank.”
Were there follow-up questions? We do not know. Would that have been
useful? According to Jennifer Rowe, writing on the Zendesk website, “In
fact, the most valuable aspect of your NPS survey results is the
open-ended feedback that you receive from your customers. From this
feedback, you can learn exactly what parts of the customer experience
can use attention and improvement.”
We don’t know the very survey schedule either. For instance, is it a
relational or a transactional survey? This difference is emphasized on
the Qualtrics
website: relational surveys are carried out on a regular basis while
transactional surveys are sent after the customer has contacted the
company.
Consequently, we have to focus on scores, markets, and timing.
Size
What is the survey size or rather the size of the sample we have
access to since data are not the real ones but have been generated from
the real distributions?
|
Responses
|
Respondents
|
First Responses
|
Last Responses
|
Aggregate NPS
|
|
5000
|
4829
|
2021-01-01
|
2021-12-30
|
12
|
The number of responses is 5,000, which is definitely enough for the
current exercise.
Is it representative of customer satisfaction on the various segments
of the different markets involved over the whole year 2021? The
importance of this issue is highlighted by Jennifer
Rowe on the Zendesk website. But we do not have that kind of
information.
The number of respondents is 4,829. Since this number is inferior to
the number of responses, some respondents have responded more than once.
Thanks to the presence of these multirespondents, we will be able to
build cohorts of respondents, for example the cohort of respondents who
responded in the first quarter and did it again in at least one other
quarter.
Our approach will be divided into three steps:
- segmentation by markets,
- segmentation by periods,
- segmentation by cohorts.
Let us first dive into market segmentation.
Market Segmentation
In market segmentation, four avenues will be explored:
- the number of responses by market;
- the NPS by market;
- the breakdown into promoters, passives, and detractors;
- the breakdown into the 10 NPS possible scores.
Responses Count
The NPS responses total 5,000. How do they break down by market?
|
Market
|
Number of Responses
|
|
MEX
|
1649
|
|
UK
|
1720
|
|
US
|
1631
|
Responses are almost evenly broken down by market, namely
approximately one third for each market, with the UK market representing
one percentage point more than each of the other two markets.
Consequently, the three markets contributed approximately with the same
weight to the aggregate NPS of 12 reached by the three markets together
(see the table just above).
The NPS responses are not the actual ones: they have just been
generated from real distributions. We do not know the actual responses
count by market and we cannot compare with recommendations such as Jennifer
Rowes’s recommendations on the Zenkesk website.
Categories
The next graph panel breaks down NPS responses by market and by
respondent category.
The MEX market has approximately the same number of passives as the
UK market. The NPS superiority of the MEX market comes mainly from the
detractors – exactly 100 detractors less on the MEX market – and
secondarily from the promoters – 36 promoters more on the MEX market.
This explains the difference in NPS: 17 on the MEX market, 8 on the UK
market.
The US market has an NPS of 10, against 8 for the UK market. In fact,
the US market has a few more promoters – 8 promoters more – and a few
detractors less – 8 detractors less. The lower number of passives along
the US side also explains one third of the difference between the two
markets.
The next graph gives the percentage of promoters, passives, and
detractors by market.
The MEX market performs better than the other two markets in both
respondent categories that matter directly: higher percentage of
promoters – with 48 % – and lower percentage of detractors – with 31
%.
The US market is almost as good in promoters but has a substantially
higher percentage of detractors – namely 37 %.
All Score Levels
Let us have a look at the breakdown of responses for each of the 10
possible scores, on each market.
There is more concentration at the extremes on the MEX market if we
just take into account the levels 0, 8, 9, and 10. Indeed, responses
whose score is one of these figures total 76.4 % on the MEX market, 66.9
% on the UK market, and 70.1 % on the US market.
This could remind us of controversies about the criteria used to
classify respondents into promoters, passives, and detractors. For
instance, Alexander Dobronte published an article on CheckMarket: “Why there needs to be a
European variant of the Net Promoter Score”. He argued that “In classic NPS scoring,
the 8 from these respondents has no weight! They are ignored. That is
why so many European companies have neutral NPS scores. What I propose
is a European Net Promoter Score variant where an 8 also counts as a
promoter and 6 as passive.”
Clifford
Lewis and Michael Mehmet in SAGE Journals also raised questions,
among others about the boundaries of the groups of promoters, passives,
and detractors, providing a long list of references.
In another direction, Maurice FitzGerald nuances the origins of NPS
differences between countries in an article titled “Does
culture affect NPS / customer survey outcomes?”. He ends his article
with: “Comparing
country scores to each other mainly wastes time and delays getting on
with improvements. Only removing the numbers from the scoring scales can
help avoid your audience making irrelevant comparisons.”
In our case, NPS differences have been noticed between the three
markets but we can hardly discriminate cultural factors from others
since we lack the kind of extensive information Maurice FitzGerald had
at his disposal. From his article, let us simply remember that
comparisons between countries are tricky.
Now, let us dive into segmentation by periods.
Temporal Analysis
Let us compute category proportions and NPS trend first by quarters,
then by months, and in the end by weeks.
Quarterly Evolution
On a quarter basis, we see
- on the MEX market: a continuous upward NPS trend;
- on the UK market: a slump in the second and third quarters followed
by a rally that nearly canceled it;
- on the US market: a growth in the second quarter followed by a
downward movement that caused NPS to fall well below the level of the
first quarter, to an even slightly negative level of -1.
Let us retrieve more detailed information by working on a month
basis.
Monthly Evolution
Switching from a quarter basis to a month basis brings more contrast
at least for the MEX and the UK markets.
On the MEX market, the first six months and the last six months show
rather different profiles. During the first six months, a sequence of up
and down movements prevailed, which of course was not perceptible on a
quarter basis. During the following six months, an upward trend took
hold, with the notorious exception of November, during which NPS fell
significantly; the fall of November was not perceptible either on a
quarter basis.
On the UK market, there are three “grapes” of results: one for the
first quarter, one for the second quarter, and one for the months August
to October. This means that, for the first six months, the quarterly and
the monthly profiles do not show substantial dissimilarities. But the
downward movement of the second and third quarters is interrupted by a
rebound in July, which did not show on a quarter basis; moreover, this
downward movement extends beyond the third quarter into October, which
did not show either on a quarter basis. The months August to October
fell to a negative level, which did not appear either in quarterly
figures. Last, the recovery noted in the fourth quarter is actually only
seen in November and December once NPS is expressed by month.
On the US market, the inverted V profile observed on a quarter basis
erodes on a month basis, producing a plateau from March to August, with
the notable exception of July, which brings about a slump. The steady
downtrend that shows in the third and fourth quarters shows with a lag
on a month basis, namely from September to December.
Weekly Evolution
Let us dive some further into volatility by visualizing the NPS
evolution on a week basis in the following graph panel.
To keep a vision of the general movement, to the weekly figures is
added a smooth curve generated by the LOESS algorithm.
In the graph panel above, weeks are identified by numbers. Conversion
to dates can easily be found on this website.
Monthly moves can correspond to continuous trends in the same
direction at week level. Let us take an example on the UK market: NPS
fell in April on a month basis; the first four weeks of April – that is
to say weeks 13 (partially in April), 14, 15, 16 – went into the same
direction with rather low NPS.
In other cases, weekly NPS can show up and downs and give indecisive
indications about the month NPS. Let’s look at the example of October on
the UK market. October has been one of the worst months with an NPS of
-4, but, on a week basis, this contrasts with week 40 – the second week
in October – being the top week of the year with an NPS of 38.
Decomposing results into respondent categories could bring some
additional insights: this is done by the next graph panel.
On the US market, NPS has been rather low in July during weeks 27,
28, and 30. When looking at the graph panel above, causes become
clearer: weeks 27 and 30 were marked by a relatively high number of
detractors; in addition, weeks 28 and – once again – 30 were
characterized by a limited number of promoters.
On the UK market, week 40 has been the top one in terms of NPS. The
graph above indicates that it has been high in promoters and low in
detractors.
After describing NPS evolution by quarter, month, and week, let us
turn to cohort analysis, which will allow us to focus again on quarterly
evolution but only among multirespondents who responded in different
quarters.
Cohort Analysis
Cohorts will be constructed on a quarterly basis. All respondents
will be regrouped in accordance with their join quarter, that is to say
the quarter of their first response.
NPS by Cohort
The next graph panel displays NPS for the cohort starting in the
first quarter and for the cohort starting in the second quarter, on the
three markets separately.
On the MEX market, the two cohorts follow similar directions. The
first cohort starts at 13. In the second quarter, it crashes and drops
to the minimum – that is to say -100. It then rebounds to the maximum –
that is to say 100 – and ends at 25 – thus above the first quarter
level. The second cohort begins at 15, jumps up to 100 and ends at 0 –
thus below the join quarter.
On the UK market, the first cohort starts at 16, falls at -100, and
then follows an upward movement up to 50. The second cohort shows an
upward trend from 4 to 83.
On the US market, moves are mostly ascending and no negative NPS
appears. Both cohorts finish at 100.
In a snapshot,
- all cohorts follow a positive path at least when comparing the end
figure to the start figure, with the exception of the second cohort on
the MEX market;
- movements are often ample and jumpy.
Since most NPS moves are so ample and jumpy, it would be interesting
to know the responses number in each cohort after the join quarter in
order to check whether NPS is representative after the join quarter.
So, let us compute the respondent retention rates by cohort and by
market.
Retention
The three tables below show, each for one market, the retention rates
of the three cohorts.
|
MEX MARKET
|
Respondents in Join Quarter
|
Retention in Join Quarter + 1
|
Retention in Join Quarter + 2
|
Retention in Join Quarter + 3
|
|
Cohort Starting in Q1
|
405
|
0.99 %
|
0.99 %
|
0.99 %
|
|
Cohort Starting in Q2
|
414
|
0.72 %
|
0.72 %
|
|
|
Cohort Starting in Q3
|
429
|
0.70 %
|
|
|
|
UK MARKET
|
Respondents in Join Quarter
|
Retention in Join Quarter + 1
|
Retention in Join Quarter + 2
|
Retention in Join Quarter + 3
|
|
Cohort Starting in Q1
|
403
|
0.25 %
|
0.25 %
|
0.50 %
|
|
Cohort Starting in Q2
|
416
|
0.72 %
|
1.44 %
|
|
|
Cohort Starting in Q3
|
474
|
1.27 %
|
|
|
|
US MARKET
|
Respondents in Join Quarter
|
Retention in Join Quarter + 1
|
Retention in Join Quarter + 2
|
Retention in Join Quarter + 3
|
|
Cohort Starting in Q1
|
420
|
0.71 %
|
1.19 %
|
0.24 %
|
|
Cohort Starting in Q2
|
388
|
0.00 %
|
0.26 %
|
|
|
Cohort Starting in Q3
|
399
|
0.50 %
|
|
|
The situation is similar on the three markets with regard to the
respondent retention rate: respondent retention is negligible.
The almost complete respondent churning can easily cause the
instability noticed in the previous section in the NPS trends by cohort.
Indeed, after the join quarters, NPS is calculated on numbers of
respondents that are almost nil. Consequently, the NPS trends by cohort
– most of which were globally on the upside – cannot be considered
significant.
Why is respondent churning so high? Neither the input data nor the
documentation provided provides a clue for further analysis.
Conclusion
Starting from a Kaggle dataset comprised of NPS survey results,
responses have been segmented by market, category, period, and cohort.
Each approach has shown insights but also limits mostly linked to input
data and related documentation.
With a view to sharing, do not hesitate to get in touch, using
contact information provided in this GitHub account.
*
* *
A project by Philippe Lambot
November 2, 2022
https://github.com/Dev-P-L